Large language models encode clinical knowledge.

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

Deciphering clinical abbreviations with a privacy protecting machine learning system.

Physicians write clinical notes with abbreviations and shorthand that are difficult to decipher. Abbreviations can be clinical jargon (writing "HIT" for "heparin induced thrombocytopenia"), ambiguous terms that require expertise to disambiguate (using "MS" for "multiple sclerosis" or "mental status"), or domain-specific vernacular ("cb" for "complicated by"). Here we train machine learning models on public web data to decode such text by replacing abbreviations with their meanings. We report a single translation model that simultaneously detects and expands thousands of abbreviations in real clinical notes with accuracies ranging from 92.1%-97.1% on multiple external test datasets. The model equals or exceeds the performance of board-certified physicians (97.6% vs 88.7% total accuracy). Our results demonstrate a general method to contextually decipher abbreviations and shorthand that is built without any privacy-compromising data.

Predicting Inpatient Medication Orders From Electronic Health Record Data.

In a general inpatient population, we predicted patient-specific medication orders based on structured information in the electronic health record (EHR). Data on over three million medication orders from an academic medical center were used to train two machine-learning models: A deep learning sequence model and a logistic regression model. Both were compared with a baseline that ranked the most frequently ordered medications based on a patient's discharge hospital service and amount of time since admission. Models were trained to predict from 990 possible medications at the time of order entry. Fifty-five percent of medications ordered by physicians were ranked in the sequence model's top-10 predictions (logistic model: 49%) and 75% ranked in the top-25 (logistic model: 69%). Ninety-three percent of the sequence model's top-10 prediction sets contained at least one medication that physicians ordered within the next day. These findings demonstrate that medication orders can be predicted from information present in the EHR.

Ensuring Fairness in Machine Learning to Advance Health Equity.

Machine learning is used increasingly in clinical care to improve diagnosis, treatment selection, and health system efficiency. Because machine-learning models learn from historically collected data, populations that have experienced human and structural biases in the past-called protected groups-are vulnerable to harm by incorrect predictions or withholding of resources. This article describes how model design, biases in data, and the interactions of model predictions with clinicians and patients may exacerbate health care disparities. Rather than simply guarding against these harms passively, machine-learning systems should be used proactively to advance health equity. For that goal to be achieved, principles of distributive justice must be incorporated into model design, deployment, and evaluation. The article describes several technical implementations of distributive justice-specifically those that ensure equality in patient outcomes, performance, and resource allocation-and guides clinicians as to when they should prioritize each principle. Machine learning is providing increasingly sophisticated decision support and population-level monitoring, and it should encode principles of justice to ensure that models benefit all patients.

Electronic Order Volume as a Meaningful Component in Estimating Patient Complexity and Resident Physician Workload.

BACKGROUND: Though patient census has been used to describe resident physician workload, this fails to account for variations in patient complexity. Changes in clinical orders captured through electronic health records may provide a complementary window into workload. We aimed to determine whether electronic order volume correlated with measures of patient complexity and whether higher order volume was associated with quality metrics. METHODS: In this retrospective study of admissions to the internal medicine teaching service of an academic medical center in a 13-month period, we tested the relationship between electronic order volume and patient level of care and severity of illness category. We used multivariable logistic regression to examine the association between daily team orders and two discharge-related quality metrics (receipt of a high-quality patient after-visit summary (AVS) and timely discharge summary), adjusted for team census, patient severity of illness, and patient demographics. RESULTS: Our study included 5,032 inpatient admissions for whom 929,153 orders were entered. Mean daily order volume was significantly higher for patients in the intensive care unit than in step-down units and general medical wards (40 vs. 24 vs. 19, P < .001). Order volume was also significantly correlated with severity of illness (P < .001). Patients were 12% less likely to receive a timely discharge summary for every 100 additional team orders placed on the day prior to discharge (OR 0.88; 95% CI 0.82-0.95). CONCLUSIONS: Electronic order volume is significantly associated with patient complexity and may provide valuable additional information in measuring resident physician workload.

Next-generation audit and feedback for inpatient quality improvement using electronic health record data: a cluster randomised controlled trial.

BACKGROUND: Audit and feedback improves clinical care by highlighting the gap between current and ideal practice. We combined best practices of audit and feedback with continuously generated electronic health record data to improve performance on quality metrics in an inpatient setting. METHODS: We conducted a cluster randomised control trial comparing intensive audit and feedback with usual audit and feedback from February 2016 to June 2016. The study subjects were internal medicine teams on the teaching service at an urban tertiary care hospital. Teams in the intensive feedback arm received access to a daily-updated team-based data dashboard as well as weekly inperson review of performance data ('STAT rounds'). The usual feedback arm received ongoing twice-monthly emails with graphical depictions of team performance on selected quality metrics. The primary outcome was performance on a composite discharge metric (Discharge Mix Index, 'DMI'). A washout period occurred at the end of the trial (from May through June 2016) during which STAT rounds were removed from the intensive feedback arm. RESULTS: A total of 40 medicine teams participated in the trial. During the intervention period, the primary outcome of completion of the DMI was achieved on 79.3% (426/537) of patients in the intervention group compared with 63.2% (326/516) in the control group (P<0.0001). During the washout period, there was no significant difference in performance between the intensive and usual feedback groups. CONCLUSION: Intensive audit and feedback using timely data and STAT rounds significantly increased performance on a composite discharge metric compared with usual feedback. With the cessation of STAT rounds, performance between the intensive and usual feedback groups did not differ significantly, highlighting the importance of feedback delivery on effecting change. CLINICAL TRIAL: The trial was registered with ClinicalTrials.gov (NCT02593253).

Scalable and accurate deep learning with electronic health records.

Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two US academic medical centers with 216,221 adult patients hospitalized for at least 24 h. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting: in-hospital mortality (area under the receiver operator curve [AUROC] across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed traditional, clinically-used predictive models in all cases. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios. In a case study of a particular prediction, we demonstrate that neural networks can be used to identify relevant information from the patient's chart.

What Happened to My Patient? An Educational Intervention to Facilitate Postdischarge Patient Follow-Up.

BACKGROUND: Following up on patients' clinical courses after hospital discharge may enhance physicians' learning and care of future patients. Barriers to this practice for residents include time constraints, discontinuous training environments, and difficulty accessing patient information. OBJECTIVE: We designed an educational intervention facilitating informed self-assessment and reflection through structured postdischarge follow-up of patients' longitudinal clinical courses. We then examined the experience of interns who received this intervention in a mixed methods study. METHODS: Internal medicine interns on a 4-week patient safety rotation received lists of hospitalized patients they had cared for earlier in the year. They selected patients for chart review and completed a guided reflection worksheet for each patient reviewed. Interns then discussed lessons learned in a faculty-led group debrief session. RESULTS: Of 62 eligible interns, 62 (100%) participated in this intervention and completed 293 reflection worksheets. We analyzed worksheets and transcripts from 6 debrief sessions. Interns reported that postdischarge patient follow-up was valuable for their professional development, and helped them understand the natural history of disease and patients' illness experiences. After reviewing their patients' clinical courses, interns stated that they would advocate for earlier end-of-life counseling, improve care transitions, and adjust their clinical decision-making for similar patients in the future. CONCLUSIONS: Our educational intervention created the time, space, and structure for postdischarge patient follow-up. It was well received by participants, and is an opportunity for experiential learning.

Caring Wisely: A Program to Support Frontline Clinicians and Staff in Improving Healthcare Delivery and Reducing Costs.

We describe a program called "Caring Wisely"®, developed by the University of California, San Francisco's (UCSF), Center for Healthcare Value, to increase the value of services provided at UCSF Health. The overarching goal of the Caring Wisely® program is to catalyze and advance delivery system redesign and innovations that reduce costs, enhance healthcare quality, and improve health outcomes. The program is designed to engage frontline clinicians and staff-aided by experienced implementation scientists-to develop and implement interventions specifically designed to address overuse, underuse, or misuse of services. Financial savings of the program are intended to cover the program costs. The theoretical underpinnings for the design of the Caring Wisely® program emphasize the importance of stakeholder engagement, behavior change theory, market (target audience) segmentation, and process measurement and feedback. The Caring Wisely® program provides an institutional model for using crowdsourcing to identify "hot spot" areas of low-value care, inefficiency and waste, and for implementing robust interventions to address these areas.

Standardized Attending Rounds to Improve the Patient Experience: A Pragmatic Cluster Randomized Controlled Trial.

BACKGROUND: At academic medical centers, attending rounds (AR) serve to coordinate patient care and educate trainees, yet variably involve patients. OBJECTIVE: To determine the impact of standardized bedside AR on patient satisfaction with rounds. DESIGN: Cluster randomized controlled trial. SETTING: 500-bed urban, quaternary care hospital. PATIENTS: 1200 patients admitted to the medicine service. INTERVENTION: Teams in the intervention arm received training to adhere to 5 AR practices: 1) pre-rounds huddle; 2) bedside rounds; 3) nurse integration; 4) real-time order entry; 5) whiteboard updates. Control arm teams continued usual rounding practices. MEASUREMENTS: Trained observers audited rounds to assess adherence to recommended AR practices and surveyed patients following AR. The primary outcome was patient satisfaction with AR. Secondary outcomes were perceived and actual AR duration, and attending and trainee satisfaction. RESULTS: We observed 241 (70.1%) and 264 (76.7%) AR in the intervention and control arms, respectively, which included 1855 and 1903 patient rounding encounters. Using a 5-point Likert scale, patients in the intervention arm reported increased satisfaction with AR (4.49 vs 4.25; P = 0.01) and felt more cared for by their medicine team (4.54 vs 4.36; P = 0.03). Although the intervention shortened the duration of AR by 8 minutes on average (143 vs 151 minutes; P = 0.052), trainees perceived intervention AR as lasting longer and reported lower satisfaction with intervention AR. CONCLUSIONS: Medicine teams can adopt a standardized, patient-centered, time-saving rounding model that leads to increased patient satisfaction with AR and the perception that patients are more cared for by their medicine team. Journal of Hospital Medicine 2017;12:143-149.

Using the Electronic Health Record to Identify Educational Gaps for Internal Medicine Interns.

BACKGROUND: An important component of internal medicine residency is clinical immersion in core rotations to expose first-year residents to common diagnoses. OBJECTIVE: Quantify intern experience with common diagnoses through clinical documentation in an electronic health record. METHODS: We analyzed all clinical notes written by postgraduate year (PGY) 1, PGY-2, and PGY-3 residents on medicine service at an academic medical center July 1, 2012, through June 30, 2014. We quantified the number of notes written by PGY-1s at 1 of 3 hospitals where they rotate, by the number of notes written about patients with a specific principal billing diagnosis, which we defined as diagnosis-days. We used the International Classification of Diseases 9 (ICD-9) and the Clinical Classification Software (CCS) to group the diagnoses. RESULTS: We analyzed 53 066 clinical notes covering 10 022 hospitalizations with 1436 different ICD-9 diagnoses spanning 217 CCS diagnostic categories. The 10 most common ICD-9 diagnoses accounted for 23% of diagnosis-days, while the 10 most common CCS groupings accounted for more than 40% of the diagnosis-days. Of 122 PGY-1s, 107 (88%) spent at least 2 months on the service, and 3% were exposed to all of the top 10 ICD-9 diagnoses, while 31% had experience with fewer than 5 of the top 10 diagnoses. In addition, 17% of PGY-1s saw all top 10 CCS diagnoses, and 5% had exposure to fewer than 5 CCS diagnoses. CONCLUSIONS: Automated detection of clinical experience may help programs review inpatient clinical experiences of PGY-1s.

Association Between a Virtual Glucose Management Service and Glycemic Control in Hospitalized Adult Patients: An Observational Study.

BACKGROUND: Inpatient hyperglycemia is common and is linked to adverse patient outcomes. New methods to improve glycemic control are needed. OBJECTIVE: To determine whether a virtual glucose management service (vGMS) is associated with improved inpatient glycemic control. DESIGN: Cross-sectional analyses of three 12-month periods (pre-vGMS, transition, and vGMS) between 1 June 2012 and 31 May 2015. SETTING: 3 University of California, San Francisco, hospitals. PATIENTS: All nonobstetric adult inpatients who underwent point-of-care glucose testing. INTERVENTION: Hospitalized adult patients with 2 or more glucose values of 12.5 mmol/L or greater (≥225 mg/dL) (hyperglycemic) and/or a glucose level less than 3.9 mmol/L (<70 mg/dL) (hypoglycemic) in the previous 24 hours were identified using a daily glucose report. Based on review of the insulin/glucose chart in the electronic medical record, recommendations for insulin changes were entered in a vGMS note, which could be seen by all clinicians. MEASUREMENTS: Proportion of patient-days classified as hyperglycemic, hypoglycemic, and at-goal (all measurements ≥3.9 and ≤10 mmol/L [≥70 and ≤180 mg/dL] during the pre-vGMS, transition, and vGMS periods). RESULTS: The proportion of hyperglycemic patients decreased by 39%, from 6.6 per 100 patient-days in the pre-vGMS period to 4.0 per 100 patient-days in the vGMS period (difference, -2.5 [95% CI, -2.7 to -2.4]). The hypoglycemic proportion in the vGMS period was 36% lower than in the pre-vGMS period (difference, -0.28 [CI, -0.35 to -0.22]). Forty severe hypoglycemic events (<2.2 mmol/L [<40 mg/dL]) occurred during the pre-vGMS period compared with 15 during the vGMS period. LIMITATION: Information was not collected on patients' concurrent illnesses and treatment or physicians' responses to the vGMS notes. CONCLUSION: Implementation of the vGMS was associated with decreases in hyperglycemia and hypoglycemia. PRIMARY FUNDING SOURCE: National Institutes of Health, the Wilsey Family Foundation, and the UCSF Clinical & Translational Science Institute.

High-Throughput Classification of Radiographs Using Deep Convolutional Neural Networks.

The study aimed to determine if computer vision techniques rooted in deep learning can use a small set of radiographs to perform clinically relevant image classification with high fidelity. One thousand eight hundred eighty-five chest radiographs on 909 patients obtained between January 2013 and July 2015 at our institution were retrieved and anonymized. The source images were manually annotated as frontal or lateral and randomly divided into training, validation, and test sets. Training and validation sets were augmented to over 150,000 images using standard image manipulations. We then pre-trained a series of deep convolutional networks based on the open-source GoogLeNet with various transformations of the open-source ImageNet (non-radiology) images. These trained networks were then fine-tuned using the original and augmented radiology images. The model with highest validation accuracy was applied to our institutional test set and a publicly available set. Accuracy was assessed by using the Youden Index to set a binary cutoff for frontal or lateral classification. This retrospective study was IRB approved prior to initiation. A network pre-trained on 1.2 million greyscale ImageNet images and fine-tuned on augmented radiographs was chosen. The binary classification method correctly classified 100 % (95 % CI 99.73-100 %) of both our test set and the publicly available images. Classification was rapid, at 38 images per second. A deep convolutional neural network created using non-radiological images, and an augmented set of radiographs is effective in highly accurate classification of chest radiograph view type and is a feasible, rapid method for high-throughput annotation.

Association Between Surgeon Scorecard Use and Operating Room Costs.

Importance: Despite the significant contribution of surgical spending to health care costs, most surgeons are unaware of their operating room costs. Objective: To examine the association between providing surgeons with individualized cost feedback and surgical supply costs in the operating room. Design, Setting, and Participants: The OR Surgical Cost Reduction (OR SCORE) project was a single-health system, multihospital, multidepartmental prospective controlled study in an urban academic setting. Intervention participants were attending surgeons in orthopedic surgery, otolaryngology-head and neck surgery, and neurological surgery (n = 63). Control participants were attending surgeons in cardiothoracic surgery, general surgery, vascular surgery, pediatric surgery, obstetrics/gynecology, ophthalmology, and urology (n = 186). Interventions: From January 1 to December 31, 2015, each surgeon in the intervention group received standardized monthly scorecards showing the median surgical supply direct cost for each procedure type performed in the prior month compared with the surgeon's baseline (July 1, 2012, to November 30, 2014) and compared with all surgeons at the institution performing the same procedure at baseline. All surgical departments were eligible for a financial incentive if they met a 5% cost reduction goal. Main Outcomes and Measures: The primary outcome was each group's median surgical supply cost per case. Secondary outcome measures included total departmental surgical supply costs, case mix index-adjusted median surgical supply costs, patient outcomes (30-day readmission, 30-day mortality, and discharge status), and surgeon responses to a postintervention study-specific health care value survey. Results: The median surgical supply direct costs per case decreased 6.54% in the intervention group, from $1398 (interquartile range [IQR], $316-$5181) (10 637 cases) in 2014 to $1307 (IQR, $319-$5037) (11 820 cases) in 2015. In contrast, the median surgical supply direct cost increased 7.42% in the control group, from $712 (IQR, $202-$1602) (16 441 cases) in 2014 to $765 (IQR, $233-$1719) (17 227 cases) in 2015. This decrease represents a total savings of $836 147 in the intervention group during the 1-year study. After controlling for surgeon, department, patient demographics, and clinical indicators in a mixed-effects model, there was a 9.95% (95% CI, 3.55%-15.93%; P = .003) surgical supply cost decrease in the intervention group over 1 year. Patient outcomes were equivalent or improved after the intervention, and surgeons who received scorecards reported higher levels of cost awareness on the health care value survey compared with controls. Conclusions and Relevance: Cost feedback to surgeons, combined with a small departmental financial incentive, was associated with significantly reduced surgical supply costs, without negatively affecting patient outcomes.